The standard error of measurement is a more appropriate measure of quality for postgraduate medical assessments than is reliability: an analysis of MRCP(UK) examinations

Background: Cronbach's alpha is widely used as the preferred index of reliability for medical postgraduate examinations. A value of 0.8-0.9 is seen by providers and regulators alike as an adequate demonstration of acceptable reliability for any assessment. Of the other statistical parameters, Standard Error of Measurement (SEM) is mainly seen as useful only in determining the accuracy of a pass mark. However the alpha coefficient depends both on SEM and on the ability range (standard deviation, SD) of candidates taking an exam. This study investigated the extent to which the necessarily narrower ability range in candidates taking the second of the three part MRCP(UK) diploma examinations, biases assessment of reliability and SEM.Methods: a) The interrelationships of standard deviation (SD), SEM and reliability were investigated in a Monte Carlo simulation of 10,000 candidates taking a postgraduate examination. b) Reliability and SEM were studied in the MRCP(UK) Part 1 and Part 2 Written Examinations from 2002 to 2008. c) Reliability and SEM were studied in eight Specialty Certificate Examinations introduced in 2008-9.Results: The Monte Carlo simulation showed, as expected, that restricting the range of an assessment only to those who had already passed it, dramatically reduced the reliability but did not affect the SEM of a simulated assessment. The analysis of the MRCP(UK) Part 1 and Part 2 written examinations showed that the MRCP(UK) Part 2 written examination had a lower reliability than the Part 1 examination, but, despite that lower reliability, the Part 2 examination also had a smaller SEM (indicating a more accurate assessment). The Specialty Certificate Examinations had small Ns, and as a result, wide variability in their reliabilities, but SEMs were comparable with MRCP(UK) Part 2.Conclusions: An emphasis upon assessing the quality of assessments primarily in terms of reliability alone can produce a paradoxical and distorted picture, particularly in the situation where a narrower range of candidate ability is an inevitable consequence of being able to take a second part examination only after passing the first part examination. Reliability also shows problems when numbers of candidates in examinations are low and sampling error affects the range of candidate ability. SEM is not subject to such problems; it is therefore a better measure of the quality of an assessment and is recommended for routine use

2020 Academic Achievement Day

2020 Oregon Academic Achievement Day book of posters and abstract

Adapting an Integrated Program Evaluation for Promoting Competency‐Based Medical Education

Educational program evaluation can improve the quality of the curriculum, instructional methods, and resources and provide useful data for making educational decisions and policies. Developing and implementing a program evaluation system is essential in competency-based medical education. The purpose of this study was to explore and establish an educational program evaluation system adapting an integrated program evaluation model to promote competency-based medical education. First, an Educational Evaluation Committee was organized, consisting of faculty, staff members, and students. The committee established an integrated program evaluation model, combining Stufflebeam’s Context, Input, Process, and Product (CIPP) model of a process-oriented approach and Kirkpatrick’s four-level model of an outcome-oriented approach. Kirkpatrick’s model was applied to the product evaluation of the CIPP model. The committee then developed evaluation criteria, indicators, and data collection methods according to the components of the CIPP model and the four levels (reaction, learning, behavior, and results) of Kirkpatrick’s model, and collected and analyzed data. Finally, the committee reported the results of evaluation to a Medical Education Quality Improvement Committee, and the results were used to improve the curriculum and student selection. To enhance the quality of education, identifying educational deficiencies and developing various elements of education in a balanced way through educational evaluation will be needed. Furthermore, it will be necessary to listen to opinions of various stakeholders, work with all members involved in education, and communicate with decision-makers in the process of educational evaluation.ope

The Effects of Clinical Practice Stress and Resilience on Nursing Students’ Academic Burnout

The high degree of academic burnout experienced during academic life indicates that job skill levels during the first year following graduation are low, and the correlation with turnover intention is high. We investigated the effects of clinical practice stress and resilience on nursing students’ burnout, and searched for factors that can prevent or control burnout. We recruited a convenience sample of 202 nursing students. Academic burnout, general characteristics, clinical practice stress, and resilience were assessed via self-reported questionnaires. The mean total score of academic burnout was 44.0 points; exhaustion was the highest at 18.5 points, inefficacy was 15.9 points, and cynicism was 9.6 points. High levels of clinical practice stress affected academic burnout (β=0.194, p=0.003), while high resilience was a factor that lowered the degree of academic burnout (β=-0.449, p<0.001). Based on our results, factors affecting students’ experiences of academic burnout were clinical practice stress and resilience. We therefore propose the implementation of a new curriculum aimed at increasing satisfaction with the major, reducing clinical practice stress, and increasing resilience, including an efficient peer mentoring program for clinical practice.ope

Educating Healthcare Professionals in Pharmacovigilance: Global Trends and Korea’s Status

This narrative review introduces global trends in pharmacovigilance (PV) education for healthcare professionals and the status of PV education in Korea. Proactive participation of healthcare professionals, including physicians, pharmacists, and nurses in reporting suspected adverse events is the main driving force for effective operation of the spontaneous adverse event reporting system database, which in turn facilitates early safety signal detection of otherwise unknown suspected adverse events. The World Health Organization recognizes PV education curriculum as a key aspect in promoting awareness of PV and adverse event reporting among healthcare professionals, and multiple studies have demonstrated that PV educational interventions for healthcare professionals have increased overall adverse event reporting. Considering the global trends in PV education, the curriculum in Korean universities still has room for improvement in promoting PV obligation among future healthcare professionals. Further research is needed to develop PV education curriculum. We suggest a three-step project for innovating PV education in Korea to meet the global PV educational standards: a survey to gauge current PV competencies among healthcare professionals, reform of current PV academic curriculum, and evaluation and fine-tuning of the reformed curriculum.ope

COIL (Collaborative Online International Learning) Program

ope